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ABSTRACT 


Machine grading techniques are becoming of greater importance 
because of the increased number of students in programming classes. 
Characteristics and limitations of automatic machine grading systems 
are proposed. A grader for introductory assembly language programming 
courses was developed. The properties of this grader are discussed 


and an example of a graded program is given. 
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I. INTRODUCTION 

A. BACKGROUND 

The capability of an instructor to teach programming courses and to 
grade student programs in classes with large numbers of students has 
been significantly enhanced by automatic machine grading systems. The 
concept of computer-controlled grading of student programs was first 
documented by Hollingsworth 1 | in 1960. Machine grading systems are 
known to exist at Rensselaer Polytechnic Institute | 1] , Stanford 
university | 2,3 |, and at universities in England and Australia. 
These grading systems provide a method of objectively grading student 
programs automatically for the instructor. By utilizing the computer 
as an extension of his teaching methods, an equal amount of attention 
in evaluating class programming work is possible. 
B. OBJECTIVE 

The objective of this thesis is to investigate the concept of 
machine grading systems and to provide a practical implementation of 
a grader for beginning assembly language students. The fulfillment 
of this objective is accomplished by an analysis of grading tech- 
niques and parameters available for evaluating a program. The second 
element of this objective is accomplished by establishing an operating 
system specifically designed for assembly language instruction at 
the Naval Postgraduate School and constructing a machine grading pro- 
gram which will operate within this system and produce graded student 
programs. Through the courtesy of Professor Andries van Dam of Brown 
University, the Brown University Student Operating System was made 
available to the Naval Postgraduate School for use in the instruction 


of assembly language. 


C. METHODOLOGY AND ORGANIZATION OF THESIS 

The methodology of this investigation was to first review previous 
work in the field of machine grading. The metrics of a program which 
can be utilized in its evaluation are discussed in Chapter II. A sum- 
mary of the essential properties of an automatic grading system is 
presented at the end of Chapter II. The description of a grading 
system and its implementation is given in Chapter III. In addition, 
grader use with student programs is discussed. The conclusions 
reached after the investigation, and the recommendations for further 
s tudy are contained in Chapter IV. 

The first step in constructing a grader system was to revise the 
structure of the Brown University Student Operating System and im- 
plement it at the Naval Postgraduate School. The Student Operating 
System was then examined for inherent characteristics which provide 
the instructor with teaching and grading capability. An appraisal 
of this system with respect to grading capabilities is presented in 
Appendix A. The other appendices contain various computer programs, 


flowcharts, and outputs relating to this study. 


II]. CHARACTERISTICS OF A GRADER 


A. REVIEW OF AUTOMATIC GRADING SYSTEMS 

A search of the literature for reports of automatic grading tech- 
niques produced several documents specifically addressing the subject. 
Perhaps the first published report of grader programs is that of 
Hollingsworth[ 1]. This grader was used at Rensselaer Polytechnic 
Institute in 1959 and was implemented on an IBM 650 computer. The 
author states that the university could not have accommodated pro- 
gramming classes of 80 to 120 students without the use of the grader. 
This early version of an automatic grader provided batch processing 
and grading of student jobs for a variety of exercises. 

Students submitted new programs and corrections to old programs 
for keypunching at the end of a class period. After keypunching, the 
corrections were filed with the old programs and all the programs, 
old and new, were arranged by exercises. The grader program deck 
was inserted with the student jobs. Considerable effort in hand 
filing punched cards was necessary in this system. 

This grader punched identification cards for student programs, 
checked results and punched either WRONG ANSWER for an incorrect 
solution or PROBLEM COMPLETE for all correct answers. 

A great deal of operator control was required to process student 
programs. Manual re-entry for overflow, invalid addresses, and 
other problems was required. A limitation of this grader was the 
fact that it processed only machine language programs. Student 


programs could modify the grader itself, causing additional problems. 


The grading system required the specification of four addresses associ- 
ated with the exercise, viz., the location of the entrance to the stu- 
dent program, the location to which the student program must exit, the 
address of variables and the address of results. 

An advantage of this system was that programming classes were not 
required to work on a single project at the same time. Since various 
exercises could be graded during one batch run, students could work on 
Several projects simultaneously if they so desired. Despite some dif- 
ficulties with this system, Hollingsworth | 1 | indicated that the grader 
had more than justified the effort in its implementation. 

Automatic machine grading programs in use at Stanford University 
are reported by Forsythe and Wirth | 2.3] . Two ALGOL grading programs 
have been used in connection with introductory ALGOL programming and 
numerical analysis courses at Stanford University since 1961]. One 
grader simply furnishes data and checks answers. The other program 
provides a searching test of the reliability and efficiency of an 
integration procedure. The major difficulties and limitations of 
the Rensselaer grader were overcome in the grading systems in use at 
Stanford. The concept of basing a grade only on the binary answer 
"CORRECT" or "WRONG" was used for beginning students. More advanced 
Students were graded by the second grader on their ability to solve 
the integration problem efficiently and with some degree of reli- 
ability. The authors claim that this grader marks a further step 
in the automation of grading because the quality of the program is 
evaluated. 

Ri. ME perry | 4 | examined the use of automatic grading programs 


for checking the practical work of students in a numerical analysis 
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course. This report provides a comparison of the Forsythe and Wirth pro- 
gram | 2 | and a grader program by Naur [5 | . Both of these graders con- 
sidered the same problem (finding the root of a given function to a given 
accuracy in a prescribed interval). 

Naur's method required that the student program be included in the 
grader program as a labelled block of coding, accessed by means of a 
Switch. The grader assigned values to variables which were global to 
the block. In this manner different problems could be presented to the 
Student algorithm. When a solution was achieved, a branch to a given 
label within the grader program was made and an examination of the re- 
sult was conducted. Further problems were then presented until the 
problem list was exhausted and the grader continued with the next 
Student program. In contrast, the ALGOL grader implemented at 
Stanford | 2 | was called by the student program. The students were 
required to write a procedure with parameters in a specific order. 

Each procedure was tested by supplying it with different sets of 
parameters and checking the solution obtained. 

One of the difficulties of grading was well illustrated in the 
work by Berry| 4]. The problem of evaluating the quality of an 
algorithm for a problem which has various methods of solution is 
exceedingly difficult. The author states that in trying to present 
a problem to which different methods of solution are available, the 
possibility of successfully grading the work presentec is remote. 

Berry points out that the demand for automatic grading programs 
in Great Britain is not expected to be substantial, since computer 
classes are smaller. One commendable aspect of graders was verified 


by both Naur [5 | and Berry| 4]. They arouse interest and provide 
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an incentive to work. Students submitting programs to this type of 
grader found the challenge of writing the best procedure an incentive 
in their work. 

The first known grading procedure for PL/1] exercises was implemented 
on an IBM System/360 model 50 computer at the Australian National Uni- 
versity in 1966 [ 6]. The decision to use automatic grading by com- 
puter of all student attempts at class problems was based on the 
following: 

1. The large volume of work required for teachers to carry out 
grading and associated record keeping. 

2. the apparent difficulty of teaching input/output sufficiently 
early in the course for students to begin writing programs. 

3. The ease with which input test values could be supplied to the 
Students and results printed under control] of a grading procedure. 

The idea of using a grading system to bypass difficult program- 
ing aspects during beginning phases of instruction was not found in 
earlier documentation of graders. Temperley and Smith | 6 | also point 
out that the use of PL/1 made possible the evaluation of results in a 
variety of formats. Test data and results could be in the form of 
arrays, structures, bit-strings and character-strings as wel] as 
scalar numeric values. 

The timeliness of this investigation is borne out by a recent paper 
On an automatic grading scheme for simple programming exercises by 
Hext and Winings| 7]. This report documents the development of the 
Basser Automatic Grading Scheme (BAGS) at the University of Sydney, 
Sydney, Australia. A major difference in the BAGS system is that it 


can handle exercises in several different languages. No special 
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action by computer operators is required with this scheme. Another sig- 
nificant departure from previously documented graders is the structure 
of the system itself. Whereas most other schemes embed their exercises 
in some larger program, BAGS is part of the standard operating system 
and its exercises are run under batch processing methods. 
The basic requirements of the program were as follows: 
" (1) It should handle exercises in ALGOL, in MINIGOL (a subset of 
ALGOL) and in KDF 9 Assembly Code. 
(2) It should not place any additional burden on the operators. 
(3) It should record every attempt at an exercise, with suf- 
ficient data for calculating a mark. 
(4) It should provide summaries on request for specified classes 
and exercises over a given period." | 7 
The method of grading in the BAGS system is unique. The assignment of 
credit for an exercise not only takes into account the mark for each 
program attempted but also the number of attempts to solve the problem. 
The maximum mark for a single exercise was set at five. One point was 
credited for each of the following: 
a. The program compiled successfully. 
b. The program ran to completion. 
c. The first answer was correct. 
d. The second answer was correct. 
e. The program ran successfully within a prescribed central 
processor time. 
The graded results were printed for each student in the form of i/j. 
This grade states that the student made i attempts at the exercise 
and that his best mark was j. If the student's program satisfied 


all five criteria the first time, a mark of 1/5 was assigned. A 


grade was assigned by the formula: 


ie 


GRADE = average of 100 xX J 


4+] 





The average was taken over all exercises being marked. 

The original version of the Brown University Student Operating Sys- 
tem included a grading system. The SOS grader has subsequently been 
eliminated from the SOS system at Brown University. In a search for 
properties of a grader it is revealing to analyze the characteristics 
of the original SOS grader to determine the cause of its demise. The 
operation of the original SOS grading system is described as follows: 

“Using appropriate supervisor calls, the student communicates 

answers and messages to the grader to indicate the problem to 

be graded (initialization), the answers and error exits to be 

generated, and to obtain special information dependent on the 

specific program being graded. In addition to communicating 

with the grader while his program is being executed, the student 

must write his program following certain conventions: 1) various 

sections of his program must bear standard labels as indicated 

in each problem assignment; 2) the data is read in the usual 

manner, but must be placed in an assigned location; and 3) the 

expected answer must be in the location and form as specified 

in each problem assignment." [ 1 | 

As can be seen by this description a great deal is required of the 
student to ensure that his program is processed properly by the 
grader. After attempting two different versions of the grader at 
Brown University, it was found that the student programs were re- 
quired to initiate almost all communication with the grading program 
and doing this made the student programs both more difficult to 
write and more difficult to make efficient, due to the rigidity of 
the communication conventions. A second, and fatal, limitation of 
this system was the inability to specify a programming problem ex- 
plictly enough to allow the student to follow grader conventions 


without actually specifying the method of solution. A review of 


two student programs which had been graded by this system and 
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discussions with the authors of the system verified these problem areas 
as the reasons for elimination of the grading feature from the Student 
Operating System at Brown University. 

It is evident that significant improvements in automatic grading 
techniques have evolved since the development of the Rensselaer grader 
in 1959. In the previous discussion of work in this field an attempt 
has been made to chronologically illustrate this evolution. Reference 
7 states that 1500 first-year students will be enrolled in program- 
ming classes at the University of Sydney in the next two terms. Under 
these circumstances, the value of grading systems seems well establish- 
ed. However, much of the documentation on this subject has concerned 
implementation of a particular grading program. The next section 
proposes certain metrics of an algorithm which might be used in a grading 
system. The effectuation of some of these ideas may not be practical 
at present. Nevertheless, with the increased interest in automatic 
grading an exploration of conceivable characteristics of a grader is 


deemed necessary. 


B. THE METRICS OF AN ALGORITHM 

The term Grader Program is perhaps a misnomer; not all such pro- 
grams assign a mark or grade to the work submitted. A grading system 
is sufficient if the program can provide enough information for a 
mark to be quickly assigned by the student's supervisor. The basis 
for the output of this information is discussed in the subsequent 
paragraphs. 

It is advisable to discuss the upper and lower bounds of a grading 
system's performance. For instance, it is a well known fact that 


there is no algorithm to determine whether or not a given procedure 
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is total. That is, a grader program cannot determine whether another 
program contains no loops. An upper limit in the performance of a 
grader program might be the class of functions which fall in this 
category. A second class of characteristics of a grader which may 
limit achievable performance, is the class of functions which are 
computable but are impractical to implement. Berry | 4] has shown that 
considerable complication results in grading a problem in which any 
one of several methods of solution might be acceptable. A lower 
bound in the performance of a grader is simply that it must pro- 

duce some information about the algorithm being tested. A program 
which recorded the trivial fact that a student program had been 
processed provides information which might be used to evaluate a 
Student's progress even though the program itself is not evaluated. 
Therefore, it can be concluded that there exists a range in which a 
grader must operate to produce the basis for the evaluation of an 
algorithm. 

The most common measure used in the evaluation of a student 
exercise is the determination of whether or not the solution is 
correct. This measure may provide a sufficient basis for a grade 
for introductory programming exercises | 1]. However, it 1S Sug- 
gested that this is dependent on the characteristics of the prob- 
lem assigned. A grader should be capable of providing information 
pertaining to the accuracy of a program result. In addition, a 
grading program should be capable of measuring this value in a 
variety of formats such as in the BAGS grading system| 7]. 

Criteria for evaluating a program which have been used are 


successful assembly, successful execution, and the number of 
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attempts at an exercise| 7]. Such criteria form an absolute judgement 
of a program in that the evaluating process does not consider the qual- 
ity of the program. 

A grader which only evaluates the results of another program does 
not provide specific information about the program itself. Qualities 
of a program which have significance with respect to grading are 
execution time, storage utilization, programming techniques, pro- 
gram logic and annotation. These characteristics form a basis for 
the measure of the quality of a program. 

The completion of a programming exercise within a sufficiently 
Short central processor time was one criteria in the BAGS method of 
grading [7]. This measure of a program can provide the instructor 
with a means of evaluating program efficiency. Application of this 
measurement is more appropriate for advanced programming classes. 

The amount of storage utilized by a program is a characteristic 
which can be used to evaluate its quality. This measure is all too 
often neglected in programming classes but has direct application in 
the field. Incorporation of this measurement in a grading system can 
be an effective means of stressing the importance of the efficient 
use of storage. 

If an instructor compared a program which contained one hundred 
ADD instructions to sum one hundred numbers with a program which used 
a single indexed ADD instruction in a program loop, the latter would 
undoubtedly be considered superior. Although programming techniques 
vary considerably, the quality measure of a program may provide the 
best criteria for evaluation of a programmer's efforts. Due to the 


fact that good programming techniques decrease execution time and 


provide efficient use of storage, this measure is more effective than 
analysis of time or storage parameters alone. The multiplicity of 
thought forms a substantial barrier to the implementation of this 
characteristic in a grading system. A specific problem could possibly 
be designed which then would be inspected for key variables in a pre- 
scribed optimum order by a grading system. It appears that the im- 
plementation of such a system would require the development of an 
extremely capable heuristic pattern matching algorithm. It is doubt- 
ful whether the effort would be justified in obtaining this measure- 
ment. 

A definition of program logic is given by Montalbano | 8]. The 
subject matter of program logic is the matching of appropriate actions 
to given conditions. For example, program logic is concerned with 
Such statements as “if a given record is an inventory adjustment, 
process the record before all other records affecting this stock 
item; if the record is a receipt, process it before any sales order 
affecting this stock item." In essence, program logic is the ef- 
fective description of computer outputs as functions of inputs. 
Techniques such as narrative description, program comments, flow 
charts, decision tables and system specification languages of the 
Iverson type are tools for expressing program logic | 8]. 

The logical structure of an algorithm is another measure which 
forms a basis for the evaluation of the program. The evaluation of 
the logic of the program is extremely difficult because of the 
variety of paths which a programmer may take to solve the problem. 
Chapter IV describes the implementation of a grader which can 


evaluate subsections of a program independently. The logical 


order in which a solution was attempted might be measured in the broadest 
sense by verifying the result of each sequential section of a program. A 
method to compare program logic descriptions such as those described in 

[ 8] would be an invaluable grading technique. At the present time an 
inspection by the instructor of program logic descriptions such as flow 
charts 1s required. 

Higher level languages have an advantage over assembler languages in 
that they possess a "self-documenting" feature [9]. The documentation 
of assembly language programs is an important area which requires 
evaluation. The annotation of a program varies from individual to in- 
dividual. The basic rules of documentation for assembly language pro- 
grams are described by opler [ 9]. It 1s doubtful if a machine process 
could automatically produce an evaluation of a student program with 
respect to these rules. This author believes that graphic methods 
may prove to be an effective way of quickly evaluating the logic and 
the documentation of a program. A correct logic flow chart might be 
Superimposed on a student flow chart on a display device. Key decision 
blocks could then be checked by the instructor. Spot checking of 
annotation can be accomplished on a graphic display device if the stu- 
dent programs are disk resident such as in the SOS system (Appendix A). 

The concept of providing automatic methods which will evaluate the 
quality of a program is in its infancy. It is expected that further 
work in this field will incorporate some of the previously listed 


metrics of a program. 


C. GRADER DESIGN CONSIDERATIONS 

The first consideration in designing a grading system should be the 
Specification of objectives. What type of information is necessary to 
evaluate the programs in a particular course? The designer must con- 
cern himself with the previously mentioned parameters of a program 
which are required to be measured. ‘ 

The second step in the design of a grader is to carefully investi- 
gate the system available for grader use. Forsythe| 3 | reports that 
the Stanford grader was possible only because of the well-designed 
BALGOL compiler with its own compiler-with-library generator, and be- 
cause a procedure called BUTTERFLY could generate relocatable machine- 
language programs. It was found that neither the IBM 7090 BALGOL 
system nor the Burroughs B5000 ALGOL 60 system were well adapted to 
the grading problem at Stanford University. 

: One of the most critical characteristics in the design of a grading 
system is the amount of involvement of the student with the grader it- 
self. The student should not be required to alter his program signi- 
ficantly to conform to grader conventions. A minimum of communications 
between the grader and the student, with maximum communication capa- 
bility between the instructor and student should be the rule in a 
grading operation. 

Another feature of a grading system which must be considered is 
the work required by the instructor to program the grader to process 
the specific problem. The grading system should include macro- 
instructions for instructor messages and general communication 
sections for the passing of answers and data. In general, instructor 


involvement with the grader should be held to a minimum. 
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The student should be able to work out solutions using his own data, 
but data should be provided by the instructor in some manner at grading 
time. Grading of the program with data from the instructor could pre- 
sent situations which had not occurred to the student. 

A grading system should be able to handle a variety of problems. 
Programming a complete grading system to evaluate a specific problem 
would be inefficient. Manual intervention by computer operators to 
process grading jobs is impractical on present large scale computing 
systems such as the IBM/360. Execution time of the student program 
Should not be significantly increased when executed for grading. 
Additional characteristics such as the ability to evaluate programs 
written in different languages, and the capability to change problem 
parameters with minimum effort should be considered in the design of 
a grading system. 

A summary of proposed grader design characteristics is presented 
in Table [. This list is a summary of features which might be con- 
sidered in the design of a grading system. 

These characteristics of a grading system have been proposed as 
the result of study of documented grading systems and also from 
limited experience gained in implementing a practical grader. Such 
features, if only achieved to some minimum degree, will provide a 
grading system which can assist the instructor greatly in programming 
instruction classes. The next chapter discussed the mechanization of 
a grading system which possesses most of these characteristics and 
is primarily designed for use in beginning assembly language pro- 


gramming courses. 
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TABLE I 


SUMMARY OF GRADER DESIGN CHARACTERISTICS 


EVALUATION CAPABILITY 


CHARACTERISTIC EVALUATED TYPE OF MEASURE 
Result of Algorithm Absolute 
Tolerance of Solution Absolute 
Successful Assembly Absolute 
Successful Execution Absolute 
Number of Attempts at Exercise Absolute 
Assembly Time Quality 
Execution Time Quality 
Number of Rcualtanauaine Executed Quality 
Storage Utilization Quality 
Program Logic Quality 
Method of Solution Quality 
Documentation Quality 


MINIMUM STUDENT COMMUNICATIONS WITH GRADER 

MINIMUM INSTRUCTOR EFFORT IN USING GRADER 

CAPABILITY TO GRADE DIFFERENT LANGUAGES 

CAPABILITY TO GRADE A VARIETY OF PROBLEMS 

AMOUNT OF OPERATOR ATTENTION REQUIRED 

EFFORT REQUIRED TO ALTER OR ADD PROBLEMS TO GRADING SYSTEM 
DATA MANIPULATION FOR PROBLEM DURING GRADING RUN 
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TIT. IMPLEMENTATION OF A GRADER 


A. PURPOSE OF GRADER 

Since the Student Operating System (Appendix A) is well-suited for 
beginning assembly language programmers, a grading program for use in 
introductory programming classes was written. The primary purpose of 
this program is to check student answers and assign a grade. It is 
designed to be used for grading small projects assigned to students 
who are just learning the language. With a grading system to handle 
numerous small projects, concentration on one particular area such as 
indexing, looping or arithmetic operations is possible and a project 
can be assigned to cover that subject. This type of grader allows 
the instructor to assign more projects and gives the student an op- 
portunity to obtain machine experience on a wider variety of assembly 
language programming aspects. This would not be permissable with a 
system which was designed for a specific problem of a more complex 
nature . 

The grader is an absolute type of system in that it operates on 
answers from the student. In addition to checking answers, com- 
munications from the instructor to the student have been included as 
an essential part in the design of this grader. In summary, the 
purpose of the grader is to allow the instructor to assign numerous 
small projects in an introductory assembly language course, to grade 
the projects and provide messages from the instructor to the student. 
By using the computer as a tool for evaluation of class work, the 
instructor can provide an equal amount of attention to appraising 


each student's progress. 
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B. MECHANIZATION OF THE GRADER 

Although the Brown University Student Operating System Grader pro- 
gram has been eliminated from the system, the structure for including 
a grader was intact in the system received. By choosing the GRADER= 
YES option at system generation time, communication links, control 
blocks, and initialization of parameters for a grader were generated 
within the SOS system. Through the use of parts of this available 
Structure and modifications to 0. elimination of some of its features, 
it was possible to concentrate efforts on the grading program itself. 

The grader operates under control of the SOS control program, and 
is called by the student when he feels that his program is correct. 
The basic structure of the grading program and its interrelation with 
the rest of the SOS system is shown in Chart Number 1, Appendix C. 
During initialization of a student's program the SOS control program 
calls the bookkeeper. The bookkeeper sets system options specified 
by the user. if the student has specified GRADER=YES on his job 
card, grader parameters are initialized by the bookkeeper and the 
grader program is called. 

A heading indicating that this is a grader run is printed and 
the SOS interpreter is then called from the grader. The SOS in- 
terpreter executes SOS machine code interpretively. The first two 
instructions of the student's program are executed setting up 
initialization of the grader for the particular problem specified 
by the student. 

After grader initialization, the interpreter continues execu- 
tion of the student's program until it is desired to have an 


answer checked. The student issues a supervisor call and indicates 
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the location of his answer. The main section of the grader checks the 
answer and assigns a grade, substitutes the correct answer if the in- 
structor so desires, prints the results and any messages from the 
instructor if the answer was not correct. Control is passed to the 
interpreter again for further execution of the student's program. 

This technique of grading the student's program during execution con- 
tinues until the final answer is graded. Return to the control program 
is then made and the next student job is processed. 

The grading program has an initialization section to initialize the 
grader for a specific problem. The main section of the grader handles 
all aspects of grading the problem. A communications section for pas- 
Sing answers and messages from the instructor is included. The grader 
is designed to grade up to five intermediate answers for a problem and 
a final answer. The capability of grading twelve different problems is 
included in the grading program. Six different answers should suffice 
as a maximum for beginning programming projects. The capability to 
grade twelve different problems allows the instructor to assign one 
problem per week during the class quarter. 

The first part of the grading program establishes addressabi lity 
with the other major sections of the SOS system and saves the return 
address to allow return to the control program at the end of the 
grading run. A grader heading is printed and the interpreter is 
given control. Exeucution of the first instruction in the student's 
program starts the initialization of the grader. The return address 
to the interpreter is first saved and the program control counter is 
incremented to point to the next sequential instruction in the stu- 


dent program. A check for student core wrap around is made and a 
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check for a proper initialization call is accomplished. A halfword 
grader control block is used throughout the program to set various con- 
ditions. The first byte of this control block is used by the system. 
The bookkeeper sets a bit on if the student has asked for a grading 
run. The grader initialization section turns a second switch on if 
grading is in process. The second byte of the grader control block 

is used by the instructor to control various functions of the main 
grading section and its use will be described in the section on use 

of the grader. 

If a grading run has been requested and initialization has not 
yet taken place, the data for the problem specified is placed in a 
general communication section. This is accomplished by the use of a 
dummy section which describes the layout of storage for the data that 
the instructor has defined for a specific problem. Symbolic names 
defined in the dummy section are then used as operands in the main 
grading section during the grading process. This method allows the 
ins tructor to separately assemble the appropriate data in a control 
aéCaien tne use the link editor to link his answer CSECT in object 
form to the system. 

Various error routines and messages are included in the grader 
initialization section. Upon completion of initialization, a mes- 
Sage to that effect is printed and execution of the student's 
program is continued. The grading program grades the result of up 
to five intermediate sec HAE which the student has used in his 
program to achieve a final answer to a specific problem. When the 
Student has reached a particular result, the main grader section 


of the grading program is called. This section first determines which 
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answer has been specified, and then branches to the appropriate routine. 
The grading routines for intermediate answers are similar except for 
correct answer usage data and the messages sent to the student. 

The intermediate answer grading routines first obtain the student's 
answer. This is accomplished by the subroutine called GETANS. The 
address of the answer in student core, which is word-oriented, is 
obtained and converted to a System/360 address. The correct answer 
specified by the instructor is then compared with the student's 
answer. If the answer is correct, a message indicating this fact is 
printed and credit for that answer is accumulated. The accumulated 
credit for each intermediate answer is also printed. If the student 
answer iS not correct, a test is made to see if the instructor has 
a message for the student which might assist him in obtaining the 
correct answer. If the instructor has included a message, it is 
printed on the student's output. The instructor has the option of 
Substituting correct data for a wrong answer if he so desires. Thus 
the subsections of a student program can be checked even if the pro- 
gram as a whole computes incorrectly. If the instructor has chosen 
this option, the correct data is moved into the student core address 
of his incorrect answer. No credit is accumulated for a wrong 
answer. A message indicating that the answer was not correct is 
printed. 

If there is just one answer for a particular problem or if a 
final solution has been reached, the final answer routine of the 
main grading section is entered. The final answer routine checks 
the solution and prints a message indicating whether it is correct 


or not. A total grade is then determined. Fifty points are given 
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for a correct final answer Each of the five intermediate answers are 
worth ten points. A grade for a problem with five intermediate answers 
and a final answer is totaled by summing the accumulated intermediate 
answer credit and the credit for the final answer. If the problem re- 
quired less than five intermediate answers, the accumulated credit for 
the intermediate answers is first supplemented to fifty and then added 
to the credit for the final answer. The assignment of these credit 
values for intermediate and final answers is arbitrary and may be 
changed by an instructor with only minor modifications to the grading 
program. The total machine grade and any final message to the student 
is then printed on the student's output. 

The next part of the grading program evaluates the efficiency of 
the student's program to some extent. The number of instructions 
executed in the student program is printed. The mean number of in- 
structions executed in previous programs for this project is supplied 
by the instructor in his answer control section. These two values 
are compared and additional credit is given to the student whose pro- 
gram required the execution of fewer instructions. A message indi- 
cating whether the student required more or less than the median is 
printed. The total machine grade with bonus credit is then printed. 
This capability is optional and is set by the instructor in the 
answer control section. 

A return to the SOS control program is then made for further 
processing of student jobs. The grader program is included in 
Appendix B. Flowchart documentation of the program is included in 


Appendix C. 
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G » USE OF THE GRADER 
This section describes the conventions which must be followed by the 
Student and the instructor to ensure proper operation of the grading 
system. Examples of graded student programs are shown in Appendix D. 
The control section established by the instructor to communicate 
answers and messages to the grader is also included as Appendix E. 
1. Student Grading Conventions 

The student is required to indicate four items to the grader. 
He must first specify that he desires his program to be graded. The 
student then indicates which problem is to be graded. When the stu- 
dent has reached a solution to the problem he must indicate to the 
grader which of the six possible answers he wants to have checked and 
the location of that answer. 

a. Calling for a Grading Run 

When the student feels that his program is ready for grading, 

he utilizes the JCL bookkeeping parameter option feature of the SOS 
system to call the grader. This is done by placing GRADER=YES on the 
job card starting in column 26. 

EXAMPLE : 

Column: ] 6 16 26 


/JOB  BACONRF SOSJOB1 GRADER=YES 





b. Specification of the Problem to be Graded 
To indicate which problem is to be graded the student 
places as the first executable instruction of his program the super- 
visor call SVC 3,0 and a second instruction DC (decimal number 0 
through 11) immediately following the supervisor call. The super- 


visor call initializes the grader for the problem specified in the 
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second instruction. For example, 1f the student desired problem number 
three to be graded, the first two instructions of his program would be 


as follows: 


Column: 





c. Specification of the Answer to be Graded 
The student must indicate to the grader which answer is 
to be graded. This is accomplished by issuing a Supervisor call. 


EXAMPLE : 


SUPERVISOR CALL SPECIFIES THIS ANSWER 
SVGe3%1 First intermediate answer 
SV Case2 Second intermediate answer 


Sloe Third intermediate answer 


SGeonA Fourth intermediate answer 


SV G4S85 Fifth intermediate answer 


SNIGES6 Final Answer 





d. Specifying the Location of the Answer 
Immediately following the supervisor call specifying 
the answer, the student places a HALT instruction with a single 


operand which is the symbolic name of the location of the answer. 
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An example of a student requesting a check of his fourth intermediate 
answer which had been previously stored in a storage location called 


ANSWER would be as follows: 


3 54 
ANSWER 





2. Instructor Grading Conventions 
The instructor must indicate four items to the grading system: 
the number of answers to be graded; the correct answers; messages to 


the student; and grader options. 


a. Specification of Answers 
To pass correct answers to the grader, the instructor de- 
fines fullword constants through the use of a macro-instruction called 


ANS. The answers will be defined in the instructor's control section. 
EXAMPLE: 


Column: 16 


Sylp=9953809/ 





The above example defines three intermediate answers and a 


final answer. Answers must be specified in order. 


b. Specification of the Number of Answers 
The number of answers to be graded is defined as a full- 


word constant in the control section. 
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EXAMPLE : 















Column: | 


10 16 
NOINTANS DC F' decimal number 0-5' 


c. Specification of the number of instructions executed. 
The instructor specifies a value for the grader to use in 
the comparison of instructions executed by the student's program by 


defining a fullword constant in the control section. 


EXAMPLE : 


Column: | 10 16 
NOX DC F'decimal number' 


d. Specification of grader options 





The grading system has three optional features. Messages 
to the student, the substitution of correct answers, and the assign- 
ment of additional credit are chosen by the instructor through the 
use of a macro-instruction called PROF. If it is desired to include 


all three features the default is chosen as follows: 


Column: 10 16 
PROF (blank operand field) 


Operands to eliminate these features are: 





NOMSG - eliminates message feature 

NODATA - correct answers are not substituted 

NOBONUS- feature to check the number of instructions 
executed and assign bonus credit is removed 


oc 


EXAMPLE: To prevent the substitution of correct answers and 


eliminate the assignment of additional credit the 


instructor writes: 










Column: 10 16 


PROF NODATA, NOBONUS 


Note: The operands may be placed in any order in the operand field. 





e. Specification of messages 
The instructor may send a message to the student whenever an 
incorrect answer is obtained and at the end of a grading run. These 
grader messages are defined through the use of a macro-instruction called 
IMSG. The instructor may define as many as six messages in the control 
section. The labels for the messages are IMESSI through IMESS5 for 
intermediate answer messages and IMESSF for a final message. Messages 


can be up to 132 characters in length. 


EXAMPLE : 


Column: ] 


IMESS4 IMSG ‘message to student’ 





Note: This message would be printed if the student's fourth 


intermediate answer were incorrect. 


The order in which these items must be defined in the con- 
trol section is shown in the example in Appendix E. The instructor 
specifies these parameters in a control section with labels which 


indicate the particular problem. GINITO indicates the first problem 
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whereas GINITII indicates the twelfth problem. The control section can 
be assembled and an object deck obtained using the System/360 ASMAD pro- 
cedure. The control section may then be linked to the SOS system by the 


linkage editor. 


D. FEATURES AND LIMITATIONS OF THE GRADER 

A comparison of the characteristics of this grader with the genera] 
properties of grading systems listed in the previous chapter is now 
presented. The SOS grader accomplishes the basic objective of de- 
termining whether an answer is correct or not and assigning a grade. 

The student involvement with the grading system itself has been 
kept to a minimum. Very little is required of the student to indicate 
the problem to be graded and correct answers. 

The student can be assisted in his programming by messages from 
the instructor. This feature allows up to six messages from the in- 
structor to the student. These messages are predefined to fit the 
circumstances of the problem solution. 

Through the use of a general communications section, ease of 
specifying the answers and messages for various problems is made 
possible. Macro-instructions for use with the answer control section 
allow the instructor to provide grader data with minimum effort. 

A feature of the grader is its flexibility in grading as many as 
Six answers for one problem and its ability to grade twelve dif- 
ferent problems during one batch run of student jobs. Students may 
work on several projects at the same time because of this capability. 

The substitution of correct intermediate answers allows indepen- 
dent grading of the subsections of the student's program. By 
evaluating each section of a program separately, a more representa- 


tive grade can be assigned. 
34 


The only feature of the grader which examines the quality of a stu- 
dent's program is the examination of the number of instructions executed. 
A better measurement of the efficiency of the program would be the 
execution time of the student program. The grader does not evaluate 
storage utilization or the number of runs submitted prior to grading. 
Inherent characteristics of the SOS system can be used by the instruc- 
tor to assign a grade based on these parameters. These characteristics 
are discussed in Appendix A. 

The SOS system uses word-oriented storage. Because of this, the 
grader is designed to evaluate fullword answers. A minor limitation 
is the fixed credit structure of the grader. However, this can be 
modified to suit a particular instructor's desires. Since the grader 
is designed for use in beginning programming classes, these limitations 
do not affect the capability to grade the type of program expected in 


introductory class work. 
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IV. CONCLUSIONS AND RECOMMENDATIONS 


A. CONCLUSIONS 

A grading system for use with the Student Operating System in begin- 
ning assembly language classes has been implemented at the U.S. Naval 
Postgraduate School. This system will significantly aid the instructor 
who must grade student exercises in classes with a large number of 
Students. 

This type of grader will provide the capability to assign more 
exercises during the academic quarter. Programming instruction can 
emphasize various aspects of the language in each assignment, thus 
allowing the student to gain more experience in assembly language 
programming. 

Since the evaluation of a programming exercise is inevitably re- 
petitive, the concept of writing a grader program which would be able 
to check other programs is both practical and useful. Documentation 
of previous grading systems has emphasized the implementation of the 
grading program. This investigation has examined the overall] con- 
cept of a grading system. Characteristics of a grader have been 
evaluated. Several of these characteristics which measure the quality 
of a program would be extremely difficult to implement at the present 
time. Despite these complexities, it is anticipated that grading 
systems of the future will concentrate on the evaluation of the qual- 
ity of a program. 

A significant quality of grading systems in general is the fact 
that the machine may be more objective in grading than the human, 


because of its notable lack of prejudice and its inability to become 


bored | 3]. 
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Grading systems generate a paradoxical situation. A system which 
is designed to minimize student involvement with the grader itself, 
invariably requires more effort on the part of the instructor. A grader 
which not only checks answers but also evaluates the quality of the pro- 
gram may require considerable additional labor by the instructor. 
Because of this situation, tradeoff between student requirements and 
instructor efforts must be fully evaluated prior to the design of a 
grading system. 

The grading system should allow the instructor flexibility in 
generating class assignments by minimizing programming work on his 
part with respect to the grader. It should be recognized however, 
that once the initial effort is expended in generating the problem, 
it can be utilized many times. 

When a grading system is to be programmed, it behooves its de- 
signer to thoroughly examine the system with which it will interact. 
The communication capabilities and limitations of the system must be 
evaluated. The system parameters such as execution time, the number 
of instructions executed, and storage limitations should be examined 
before the initial design of a grading system. In this manner, 


various. existing facilities may be used in the grading system. 


B. RECOMMENDATIONS 

It is strongly recommended that the grading system be used at the 
Naval Postgraduate School. It represents a powerful aid to assembly 
language programming instruction. 

Recommendations for augmentation of the grader are: 

(a) Inclusion of the standard data option of the Student Opera- 


ting System to be used in conjunction with the grader. 
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(b) Revision of the Student Operating System statistics keeper so 
that it can produce a grade book based on results obtained 
from the grader. 

(c) Modification of the grader by replacing student supervisor 
calls to the grader with simple Student Operating System macro- 
instructions. 

(d) Provide the Student Operating System with a meaningful timer 
and revise the grader to utilize execution time as a measure 
of the efficiency of student programs. 

(e) Assignment of these recommendations to students as suitable 
term projects in an assembly language programming class. 

Very little work has been done in developing effective techniques 

to automatically evaluate the quality of a program. It is recommend- 
ed that further investigation of both the theoretical and the practi- 


cal phases of this area of grading be conducted. 
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APPENDIX A 
The Student Operating System 


A. PROGRAMMING INSTRUCTION AT THE POSTGRADUATE SCHOOL 

Students taking a beginning Computer Science course at the Naval 
Postgraduate School have ample opportunity for actual machine ex- 
perience. This is not the case for some universities where cost 
considerations take precedence and research work usually justifies 
the existence of the university computer center. 

To add credence to the idea of efficiently instructing programming 
courses, an indirect measure of the amount of programming instruction 
at the Postgraduate School was obtained. This measure is the number 
of WATFOR jobs processed. WATFOR is an acronym for the Waterloo FOR- 
TRAN compiler. This system has been used extensively for introductory 
FORTRAN programming classes since October, 1967. 

Table II presents the Naval Postgraduate School Computer Facility 
usage data. Approximately forty percent of the total number of com- 
puting jobs submitted in the past eighteen months have been WATFOR 
jobs. This is an indication that a primary use of the Computer Facil- 
ity iS programming instruction. Many of the jobs listed in the 
STUDENT category of Table II are also programming instruction class 
work. Faculty programming jobs comprise 8.75 percent of the total 
and are assumed to be primarily associated with research studies. 

This evidence indirectly substantiates the fact that a major use of 


the Computer Facility is programming instruction. 
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B. THE ORIGINAL STUDENT OPERATING SYSTEM 

A system which can be used for assembly language instruction is the 
Brown University Student Operating System [ 10]. This student operating 
system provides an assembler which produces code for a simplified 
machine, an interpreter which simulates the simplified machine and a 
control program for storage manipulation, editing of the student's pro- 
gram and statistics gathering. Wile [10 | documents the capability of 
this system to provide assembly language teaching and processing at 
costs comparable to other university algebraic compilers used for 
student programming instruction. 

The Student Operating System (SOS) offers several advantages to the 
instructor who wishes to utilize the IBM System/360 in an assembly 
language class. The feature of changing basic parameter settings at 
System generation time or taking default options allows the instructor 
to change the capabilities of the entire system for particular pro- 
gramming projects. All of the system is in core at all times and set 
up for each student job requires less than twenty machine instructions. 
SOS utilizes a simplified job control language, thus practical ly 
eliminating the difficult task of explaining the use of and reason 
for the complex, but extremely powerful, general purpose JCL required 
by the System/360 Operating System. Student jobs are cataloged on 
a disk; editing is done directly on the cataloged programs, thus 
introducing the student to the concept of text editing and library 
maintenance as well as reducing the volume of cards which the computer 
1s required to read. 

The SOS machine and assembly languages were designed to eliminate 


some of the more difficult programming concepts inherent in the 
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System/360 machine and assembly languages | 10]. However, the SOS pro- 
cessing languages provide a compatible introduction to the related 
System/360 languages as the SOS assembly language mnemonics are, in 
general, a subset of the System/360 mnemonics. Arithmetic and logical 
operations are performed in a similar manner to System/360 operations. 
Data is in two's complement form and operations are performed in 
registers producing results and interruptions similar to 360 instruc- 
tions. 

The instructor may view the elimination of base/displacement ad- 
dressability considerations, variable length instruction and data 
formats, floating point, packed decimal and character format con- 
versions, relocatability, control and dummy section capabilities, 
and condition code testing from the assembly language repertoire as 
being too restrictive for the proper introduction of assembly language 
programming to the student. However, it is the opinion of this author 
that provisions included in the SOS system for indexing, indirect 
addressing, register addressing, overflow testing, and all arith- 
metic and logical operations are adequate to accomplish the objective 
of assembly language instruction at this school. A reasonable state- 
ment of this objective is to provide the student with an under- 
Standing of the structure and organization of a computing machine. 

It is not the intent of this study to document in detail the Student 
Operating System and its language. Specifications for the system 


and a description of its use may be found in Ref. 11. 


C. IMPLEMENTATION OF THE SOS SYSTEM 
The SOS System was obtained from Brown University in October, 


1968. Documentation of the system received was negligible. System 
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generation was accomplished by issuing a single macro instruction (SOS); 
all program segments were included as lower level macro calls. This 
version was unsuitable for practical use in implementing a grader pro- 
gram within the system or for making changes to the system in general. 
The "SOS" macro included system default parameters. Required global 

set symbols were included in this macro. The main section of the "SOS" 
macro was the SOS processor common area. This section called the eight 
major macros in the system. The system assembled as the single macro- 
instruction "SOS" with this structure. 

The IBM System/360 assembly language includes the COPY instruction. 
This instruction obtains source language coding from a library and in- 
cludes it in the program currently being assembled. The assembler 
inserts the requested coding immediately after the COPY instruction 
is encountered. This instruction was utilized to provide a system 
structure which allowed manipulation of each of the eight major 
sections separately. The macro structure of each of the previously 
mentioned macros was removed and all global set symbols were placed 
in a section named SOSGBL which was then used as copy text in the 
SOS macro. Keyword parameters in the SOS macro were modified to 
make them compatible with the revised structure. In addition, the 
SOS macro was shortened to include only the necessary set symbol 
instructions. The original SOS processor common area was modified 
to call in the eight major program sections of the system by the 
use of COPY statements instead of the previous macro-expansion 
method. This section of coding was renamed SOSMAIN. The modified 


version of the system was generated on 7 February 1969. 
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D. SOS CHARACTERISTICS FOR STUDENT EVALUATION 

Several inherent characteristics of the SOS system interrelate with 
the concept of grading a student's programming work. In describing the 
operation of a grader in use at Stanford University, Forsythe and Wirth 
[ 2] state that it iS inappropriate to grade beginning students on the 
execution time of their programs or to evaluate the amount of storage 
used by their programs. Certainly the novice programmer should not be 
severely restricted in his programming efforts by these parameters. 
However, he should be made aware of their existence and thus eliminate 
the breeding of poor programming habits at the outset of instruction. 
By using the capability of varying system parameters in the SOS system 
the instructor may impose restrictions which will require the student 
to consider such aspects of programming as storage utilization, the 
number of instructions executed, register usage, and the number of 
program runs to achieve a solution. 

The capability of imposing restraints on the student is essentially 
a method of evaluating his ability to program within the restricted 
environment. It is not an absolute grading mechanism, but rather a 
circuitous method of examining his programming ability. The features 
of the SOS system which provide this capability are discussed in the 
subsequent paragraphs of this section. 

The highest configuration of SOS core the instructor wishes his 
Students to use can be set by the keyword parameter MAXCNF = 
(1/2/..../8) at system generation time or by the bookkeeping parameter 
CONF = (1/2/..../8) at execution time. The student core size is then 
set to the value specified times 512 words. An instructor could 


generate a problem whose solution required approximately 1000 words 
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of storage and then specify in the assignment that the bookkeeping param- 
eter CONF = 2 must be used on the student's job control card. If the 
student solved the problem within a 1024 storage environment, he could 

be given additional credit. The poorer student could still achieve the 
solution but would use the default parameter of 8 (4096 words). The 
actual configuration used is printed on the first page of the student's 
output by the bookkeeper. The ability to vary student core size is 
beneficial to the student because he can be made aware of real-world 
machine limitations at the start of his programming experience. 

One means of student evaluation is to limit the number of program- 
ming runs that a student may submit prior to final grading of a project. 
This method insures that the student will carefully think out each step 
of his program and thoroughly review each instruction before submitting 
it for execution. 

The SOS system provides the capability to record the number of runs 
a student submits for execution. If the system is used properly, stu- 
dents will be required to catalogue their programs in the student job 
library on a disk. All changes to the program are then made through 
the use of the SOS editing feature. On the first page of the student's 
output under the data card listing a message is printed "NEXT RUN IS 
NO.----." This provides the instructor with a means of evaluating the 
ability of the student to achieve a solution to the problem in as few 
runs as possible. Again this characteristic of thesystem does not 
pass an absolute judgement on a student's solution to a programming 
project. It does however, provide a better means of evaluating the 


overall potential of the student. 
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A third built-in grading characteristic of the SOS system is the 
capability to vary the maximum number of allowable executable instruc- 
tions. The primary purpose of this feature is to prevent infinite 
looping. The maximum number of allowed executable instructions may be 
set at system generation time by the keyword parameter SOSMAX = 
( , , maximum number of instructions) or at execution time by the 
Student through the use of the bookkeeping parameter INSC = (decimal 
number) on the student's job card. This parameter is evaluated 
directly by the Grader and is discussed in Chapter III. 

In summary, the major capabilities of the SOS system with respect 
to student program evaluation are storage utilization flexibility, the 
capability to record every attempt at an exercise, and variability of 
the maximum allowable number of executable instructions. Imaginative 
and prudent application of these features of the SOS system by an 
instructor should motivate the student toward good programming prac- 
tices and at the same time allow a broader evaluation of the true 


capability of the student. 
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APPENDIX B 


The Grader Program 
The Grader Program is shown in assembled form. The Grader Program 
is assembled with the Student Operating System when the keyword param- 


eter GRADER=YES is chosen at system generation time. 
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APPENDIX C 


Flowchart Documentation of the Grader Program 
The flowcharts which document the various sections of the Grader 


Program are the fol lowing: 


Flowchart Number Title 
ile The Basic Grading Routine 
Zz Grader/SOS Interface and Grader 
Initialization 
5, . Intermediate Answer Grading Routine 
4, Final answer Grading Routine 
o.. Routine for Accumulating Intermediate 


Answer Credit 
6. Routine to Obtain Student's Answer 


ye Translation and Printout of 
Machine Grade Routine 


8. Return Routines 


9. Total Machine Grade Routine 
(First Section) 


10. Total Machine Grade Routine 
(Second Section) 


hie Instructions Executed Routine 
12. Structure of General Communications 
Section 
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An example of a program with the grader off is first shown. 


Examples of graded programs are then given. 
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